Connected processing to loading the dataset. #11
Conversation
…ntation to explain how it works. added random seed
| transformations=TRANSFORM_SOME, | ||
| ) | ||
| test_loader.load(datapath=TEST_DATA_MRC, datatype=DATATYPE_MRC) | ||
| assert not test_loader.dataset.normalise |
There was a problem hiding this comment.
we could add a test to make sure the transformation is happening, e.g. that the output dataset (for an example data point) is different if you pass a transformation to passing none.
There was a problem hiding this comment.
Latest commit addresses this!! I hope that's ok, made a debug flag that doesn't shuffle the paths so we can test the processing is happening.
| @@ -127,3 +129,33 @@ def test_get_loader_training_fail(): | |||
| torch_loader_train, torch_loader_val = test_loader.get_loader( | |||
| split_size=1, batch_size=64 | |||
| ) | |||
|
|
|||
There was a problem hiding this comment.
Also, it will be good to test the data loader returning what we want, I think here we have only tested that the class variables are correct (which is great!) but we won't catch things like the data loader not returning the right size of data, or selecting correctly the labels.
There was a problem hiding this comment.
that's a solid point! I checked it but realised it was only on a local function, I'll make that into assertions here
…g flag for dataloader so that there's no shuffle in the paths to test this out
|
Took a little while but I think I've addressed everything, including the rescale bugs, testing before/after processing |
|
Looks good @mooniean! Just don't forget to add docstrings to document variables, but that can be in another PR to not stop progress. |
added processing when transformations are present.
need to add documentation to explain how it works.
added random seed
Addresses